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Abstract 

NASA’s Aviation Safety Program (AvSP) develops and advances methodologies and technologies to improve air 
transportation safety. The Safety Analysis and Integration Team (SAIT) conducts a safety technology portfolio 
assessment (PA) to analyze the program content, to examine the benefits and risks of products with respect to 
program goals, and to support programmatic decision making. The PA process includes systematic identification of 
current and future safety risks as well as tracking several quantitative and qualitative metrics to ensure the program 
goals are addressing prominent safety risks accurately and effectively. One of the metrics within the PA process 
involves using quantitative aviation safety models to gauge the impact of the safety products. This paper 
demonstrates the role of aviation safety modeling by providing model outputs and evaluating a sample of portfolio 
elements using the Flightdeck Automation Problems (FLAP) model. The model enables not only ranking of the 
quantitative relative risk reduction impact of all portfolio elements, but also highlighting the areas with high 
potential impact via sensitivity and gap analyses in support of the program office. Although the model outputs are 
preliminary and products are notional, the process shown in this paper is essential to a comprehensive PA of 
NASA’s safety products in the current program and future programs/projects. 
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Introduction 

The Aviation Safety Program (AvSP) of NASA is a “focused” program to develop and advance methodologies and 
technologies (referred to as products) to improve air transportation safety. The AvSP is comprised of three research 
projects: Vehicle Systems Safety Technologies (VSST) Project, System-wide Safety Assurance Technologies 
(SSAT) Project, and Atmospheric Environment Safety Technologies (AEST) Project. These projects are focused on 
the AvSP’s top ten technical challenges (TCs) in the areas of accident prevention and mitigation, aviation system 
monitoring, and modeling. As of 2013, there were 46 safety products being developed within these TCs in the 
AvSP technology portfolio. The safety products include innovative algorithms, tools, concepts and technologies for 
improving the overall safety of vehicles and systems operations in the future. To examine the benefits and risks of 
these products to the program goal, NASA Systems Analysis and Methods (SAM) group within the Safety Analysis 
and Integration Team (SAIT) of the AvSP led a portfolio assessment (PA) to support the Program Director’s 
decision making. As part of the PA process, the SAIT employs both quantitative (modeling, programmatic and 
expected implementation costs) and qualitative (technical development risk, implementation risk, technical readiness 
level) metrics alike (Jones et al., 2010). 

The paper is comprised of two sections. First, an overview of the FLightdeck Automation Problems 
(FLAP) model, a high level systems-integrated model for examining safety issues due to automation, will be given. 
The FLAP model is one of a series of three aviation safety models that were developed to provide a quantitative 
safety metric to support the AvSP PA (Luxhoj et al., 2012). The other two models are the in-flight Loss-of-Control 
Accident Framework (LOCAF) model (Ancel et al., in press) and the runway safety (RUNSAFE) model (Green, 
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2014). The second section illustrates the modeling effort’s contribution to the overall PA process using the FLAP 
model and a small set of sample notional products. The outcomes of the modeling effort presented in this paper are 
integrated into the overall technology PA process by combining other metrics of interest which provides valuable 
information for decision making and communication. 

FLAP Model 

This section provides a brief outline of the FLAP model which was developed using Bayesian Belief Networks 
(BBNs) and modeled with the Hugin Software (Hugin Expert, 2013). A BBN is a directed acyclic graphical 
representation of a network which contains a set of nodes representing causal factors connected via links, 
designating the causal dependencies. The modeling process including the extensive literature review, modeling 
approach, boundary conditions, assumptions and data population is provided in detail in Ancel & Shih (2014) and 
Luxhoj et al. (2012). Recent studies identified vulnerabilities in pilot use of, and interaction with automation systems 
as well as manual flight operations related to pilot complacency and heavy reliance on automated systems (FA A, 
2013 a, 2013b). Consequently, the focus of the FLAP model is on the effects of increased complexity and reliance on 
automation systems in transport category aircraft accidents and incidents. Following subsections provide the node 
definitions and interactions throughout the model. 

LI Level: Latent Organizational Factors 

The bottom three nodes include the LI level organizational latent factors such as the Regulatory Body, 
Manufacturer, and Operators/Airlines (Exhibit 1). The Regulatory Body node represents deficiencies within the 
regulatory process in both aircraft certification and flight standards of commercial transport operations (FAA, 2013). 
The Regulatory Body node is linked to five other nodes: Manufacturer, Training, Operators/ Airlines, Automation 
Design, and Automation Interface. The Manufacturer node represents large aircraft manufacturers as well as those 
companies which manufacture automation systems or avionic equipment. The node covers deficiencies in 
automation design philosophy and approach, over-automation, and standardization and also cultural diversity that 
could eventually lead to human factors issues (FAA, 1996). The Manufacturer node influences characteristics of 
Automation Design, Automation Interface, and also affects Training and Policy /Procedures nodes (FAA, 1996). 
The Operators/Airlines node delineates the organizational aspects of corporate airlines as causal or contributing 
factors in automation accidents/incidents. The organizational deficiencies can trickle down and materialize as 
Training (Parasuraman & Riley, 1997) or Policy /Procedure (FAA, 1996, p. 65) issues. Also, lack of adequate 
supervision and management guidance can result in Adverse Physiological or Mental States in flightcrews. Finally, 
links between the Operator/ Airlines and Manufacturer nodes represent unrealistic airline expectations or 
requirements imposed on the manufacturers, driven by economic motivation and operational efficiency. 

L2 Level: Latent Underlying Factors 

The second latent layer includes underlying factors affecting both automation systems and airline operations. These 
factors are Training, Policy /Procedures, Automation Design, Automation Reliability, and Automation Interface. The 
Training node includes deficiencies associated with inadequate training, generally associated with the common 
practice of “on-the-job” training (Orlady, Orlady, & Lauber, 1999; Sarter, Woods, & Billings, 1997). This node is 
connected to Flight Skills Degradation (designates the basic stick & rudder training), Awareness/Monitoring, 
Understanding/System Knowledge, and Crew Resource Management (CRM) nodes, representing respective training 
constituents. The Policy /Procedures node covers deficiencies associated with inappropriate flightcrew guidance. For 
instance, procedures inconsistent with manufacturer recommendations, or incorrect modification of procedures for 
economic or fuel saving reasons are examples of issues covered in this node (FAA, 1996, 2013). The 
Policy /Procedures node is connected to the Flight Skills Degradation, Trust/Reliance, Understanding/System 
Knowledge, Training, and CRM nodes, representing cases of inadequate operator guidance on automation usage and 
aircraft control. The Automation Design node encompasses issues within the automation system design process 
including system, hardware, and software designs from preliminary phase to flight hardware including assumptions, 
requirements, testing/debugging, implementation, verification and validation, quality assurance, etc. The 
Automation Design node is connected to all the downstream nodes in the automation related Al level nodes since 
improper planning and execution of automation requirements can result in failures and unexpected behavior 
throughout the system. The Automation Reliability node primarily affects the flightcrew’ s perception where highly 
reliable automation systems inherently increase reliance on automation. However, deficiencies found in automation 
design could potentially result in a higher number of failures, which in turn, affects the perceived system reliability 
by pilots (FAA, 1996), therefore connecting the Automation Reliability node to the Trust/Reliance node. 
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Exhibit 1. FLAP Model 
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The final node in this level is the Automation Interface. The automation interface deficiencies are linked to 
degraded flightcrew situational awareness, pilot saturation and/or confusion (FAA, 1996). This node contains issues 
related to determination of the characteristics of visual, auditory, and tactile alerting systems as well as alert 
categorization issues (Ancel & Shih, 2014; Veitengruber, 1978). The Automation Interface node is linked to three 
causal factors; Trust/Reliance, Awareness/Monitoring (Sarter et al., 1997), and Understanding/System Knowledge 
(FAA, 2013) as shown in Exhibit 1. 

L3 Level: Latent Flightcrew Factors 

The third level of latent causal factors consists of Flightcrew (FC) Experience/Background, Trust/Reliance, Flight 
Skill Degradation, and Under standing/ System Knowledge nodes. The FC Experience/Background node 
distinguishes the varying flightcrew personal factors including experience level (seasoned vs. young pilots), training 
background (military vs. civil aviation), and operational environment. The Flightcrew Experience/Background node 
is linked to Flight Skills Degradation, Understanding/ System Knowledge, and flightcrew Trust/Reliance nodes 
(Parasuraman & Riley, 1997; Sarter et al., 1997). The Trust/Reliance node includes flightcrew complacency and 
inappropriate confidence level assigned to autoflight systems. The Trust/Reliance node is a parent causal node for 
three nodes; namely, Awareness/Monitoring (Sarter et al., 1997), Flight Skills Degradation, and Decision Deficiency 
(Parasuraman & Riley, 1997). The Flight Skills Degradation node contains erosion of manual flight skills (e.g. 
basic stick-and-rudder capabilities, instrument scan, etc.) due to continuous operation of autoflight systems and lack 
of practice (FAA, 2013a; Orlady et al., 1999). The Flight Skills Degradation node is connected to both the Flight 
Anomaly and Final Recovery nodes to represent the cases where degraded skills can cause flight anomalies or 
unsuccessful flight recovery. The Understanding/System Knowledge node refers to issues related to flightcrew 
knowledge of aircraft systems or presence of gaps in their mental model of the system (e.g. autoflight modes, flight 
management computer, system couplings). This node is connected to the Decision Deficiency node since issues 
with pilot Understanding/System Knowledge is one of the primary sources of errors captured in the Decision 
Deficiency node. Pilot understanding is also linked to Automation Surprise and flightcrew System 
Awareness/Monitoring nodQS (FAA, 1996). 

Al Level: Active Factors/Automation Triggers 

The first active layer includes the Operating Environment node, the Hardware/Software Failure node, five 
automation function nodes, and the Automation Issue node. The Operating Environment node provides external 
causes that potentially affect the operation of hardware/software (HW/SW) either by disrupting sensor outputs or by 
damaging aircraft systems directly. The HW/SW Failure node includes all glitches and malfunctions of the systems 
that were not anticipated by the designers, including malfunctions of antennas, sensors, etc. that provide information 
to the automation systems downstream nodes. The Performance Systems node includes issues associated with the 
performance function of the flight management system (FMS) (e.g., weight and balance, fuel weight, take off 
reference data, etc.) (Walter, 2001). The Warning & Monitoring Systems include automated warning systems such as 
aircraft configuration, monitoring of aircraft systems and presence of environmental threats due to both faulty design 
and HW/SW failure (Orlady et al., 1999). The Navigation Systems node covers the components and systems used in 
navigation including all precision approach system components. The Flight Control Systems node encompasses all 
the systems involved in automatic flight within the FMS and implementation of inputs via flight control surfaces. 
Finally, the Communication Systems node includes data link and surveillance systems. Systems like the Aircraft 
Communications Addressing and Reporting System, telemetry, communication radios, satellite links, telemetry, and 
ADS-B/C are included in this node (Walter, 2001). The Automation Issue node provides the probability of an 
automation system exhibiting malfunction or failure, stemming from any of the five functional systems described 
above. Besides failures, inconsistent or unexpected automation system behavior itself can also be the root of 
flightcrew confusion or decision deficiency and it is represented within the Automation Issue node. Unexpected 
automation behavior which is captured in the Automation Issue node is linked to Automation Surprise and A/C 
System Distraction nodes (Sarter et al., 1997). 

Al Level: Active Factors/Flightcrew Triggers 

This level considers external and internal sources of distraction as well as psychological and physiological aspects of 
FC performance, in an approach similar to Reason’s model of accident causation. The flightcrew triggers level is 
divided into two sections: FC active factors prior to the flight, and FC active factors during the flight. 

Flightcrew Active Factors - Pre-Flight. The causal factors in this section help determine the FC readiness before 
the flight takes place. The Adverse Physiological States node includes physical fatigue/lack of sleep, medical illness. 
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and physiological incapacitation in general. The node is connected to the FC Preconditions node and Dynamic FC 
Conditions node, which then connects to the Awareness/Monitoring node that is greatly affected by the presence of 
physiological issues. The Adverse Mental States node includes complacency, distraction, get-home-itis, misplaced 
motivation, mental tiredness, distraction, confusion, depression, and/or alcoholism. The Crew Resource 
Management (CRM) node includes deficiencies such as communication skills and coordination that take place 
among the flightcrew as well as between other entities before, during, and after the flight (Ancel & Shih, 2012). The 
CRM node is linked to both FC Preconditions and Dynamic FC Conditions, indicating issues associated with the 
pre-flight briefing as well as in-flight communication and coordination, respectively. The Flightcrew Preconditions 
node is an aggregation node that considers flightcrew adverse physiological and mental states as well as CRM 
deficiencies and determines the fitness of the flightcrew for the upcoming flight. This node is linked to the Dynamic 
Flightcrew Conditions node, providing a baseline which is then updated by considering the presence of distractions 
throughout the flight. 

Flightcrew Active Factors - In-flight. This section provides the model with the updated FC conditions that are 
present during the flight by considering several sources of distractions. These distractions are divided into two 
categories; aircraft system related distractions and general distractions. The Aircraft (A/C) Systems Related 
Distractions node provides the probability of flightcrew to be distracted by the presence of a) Automation Issues and 
b) System Component Failure SCF. Distractions stemming from troubleshooting autoflight system 
anomalies/behavior as well as reprogramming the FMS are captured in this node. The Distractions node takes into 
consideration all other major sources of mental disturbance that potentially result in fixation or absorption. The 
causal factors considered are fourfold; Traffic, ATC, On-board Personnel, and Weather. The presence of traffic, 
frequent changes in the flight trajectory, cabin crew interference, presence of icing, fog/visibility issues, wind, etc. 
were identified as sources of distraction (CAST, 2008). The only output of the A1 Level is the Dynamic FC 
Conditions node which is connected to Flight Skills Degradation, Decision Deficiency, and Awareness/Monitoring 
nodes. 

A2 Level: Active Factors/Mishap 

The A2 active failure level includes nodes associated with flightcrew System Awareness/Monitoring, Decision 
Deficiency, and Automation Surprise which lead to Flight Anomaly, and Final Recovery. The 
Awareness/Monitoring node comprises insufficient system awareness, defined as the inability of a supervisor to 
track and anticipate the behavior of a) the automation system variables and controls, b) aircraft state and flight 
parameters, and c) operating environment (Sarter et al., 1997). This node is linked to the Decision Deficiency and 
Automation Surprises nodes (Sarter et al., 1997). The Decision Deficiency node includes all cognitive errors made 
by the flightcrew such as mode selection/confusion error, FMS programming, checklist/procedural errors, 
misdiagnosis of faults, etc. The Decision Deficiency node influences three nodes; Automation Surprise, Flight 
Anomaly, and Final Recovery. The Automation Surprise node defines cases where the operator is surprised by the 
automation systems, unable to comprehend its current behavior or estimate future occurrences. The presence of 
automation surprises is one of the prominent causal factors for the Flight Anomaly node, in which the flightcrew 
recognizes that the aircraft is outside its flight envelope or air traffic control (ATC) interventions. The Flight 
Anomaly node designates any departure from the intended flight plan or safe flight envelope that qualifies as an 
incident (e.g. deviation in altitude, speed, position, etc.) which may potentially develop into stall, loss-of-control, 
over-speed, loss of separation, controlled flight into terrain, etc. The Final Recovery node refers to the ability of the 
flightcrew to recover from an abnormal flight condition defined in the Flight Anomaly node. Given that the model 
simulates an accident and incident environment the Final Recovery node plays a decisive role in whether the 
incident turns into an accident (Ancel & Shih, 2014). 

FLAP Model Preliminary Results 

As shown in Exhibit 1, the model output nodes are Automation-Related Event Probability and Automation-Related 
Incident/ Accident Probability. The Automation-Related Event Probability node reflects the ratio of automation- 
related events among all accidents and incidents. The uncalibrated preliminary results of the FLAP model indicate 
that around 78% of all U.S. based accidents and incidents in today’s commercial aircraft are related to pilots’ 
automation usage. The remaining 22%> of these events are not tied to automation. The other output node, 
Automation-Related Incident/ Accident Probability, provides the accident to incident ratio using the Final Recovery 
node. The preliminary results indicate that that around 2.7% of all automation related events would result in an 
accident, and the remaining 97.3% of them will be considered as incidents. 
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Besides the output nodes, all other causal factors nodes can all be accessed individually. The preliminary 
results show that around 80% of all events involve flightcrew Decision Deficiencies, whereas 72% of the cases were 
tied to failure in situational awareness {Awareness/Monitoring node). The Flight Skills Degradation node plays a 
role in around 70% of all the cases and the Automation Issue node which includes malfunctions and unexpected 
automation behavior is present in 50% of the cases. It is important to note that these values do not represent the 
final values of the modeling effort, nor are they intended to be used in flightdeck automation, policy or decision 
making processes. The next section provides an overview on how the model data is employed. 

AvSP Product Insertion Process 

The technology PA is initiated with the insertion of methodologies and technologies (or products) developed within 
the AvSP projects onto the baseline FLAP model. The product placements and assessments are performed by subject 
matter experts (SMEs). First, the SMEs review all provided documentation and descriptions regarding the products. 
Next, they consider the products’ primary and secondary mitigation effects and link the products to the affected 
causal factor nodes. Finally, the SMEs provide updated conditional probabilities for each node considering the 
products’ impact on the risk reduction. Following the product insertion process, the location of the products and the 
model construct are verified by product developers and external expert panels to ensure proper placement, as part of 
the verification process. The rectangular nodes shown in Exhibit 2 represent notional products and the arrows 
designate the affected nodes only within the top part of the model. The five products shown in the exhibit are used 
as examples in the upcoming sections to demonstrate the PA process. 

Exhibit 2. Notional Product Insertion 



Technology Portfolio Assessment Process 

The main purpose of developing high-level, system-integrated aviation safety models is to provide a quantitative 
metric to support AvSP management in technology PA. As previously discussed, AvSP portfolio elements are 
evaluated against several metrics including the portfolio’s expected impact on historic and future safety risks in Part 
121, 135 and 91 operations as well as NASA’s high-level goals and objectives (i.e. National aeronautics R&D 
programs and the JPDO safety objectives). Besides portfolio elements’ impact on safety risks, their technology 
readiness level (TRL), technical development risk (TDR), implementation risk (IR), cost (both expected 
implementation cost and programmatic cost) are also within the scope of the SAIT PA process (Jones et al. 2013). 
Although demonstrated within the AvSP PA process, the modeling effort’s contribution described in the next sub- 
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sections could be used by future NASA programs/projects (e.g. the new Airspace Operations and Safety Program, 
AOSP, or its Safe Autonomous Systems Operations Project, SASO) as well as other organizations (e.g. FAA 
NextGen). Following sections highlight the model results including relative risk reduction, sensitivity analysis and 
gap analysis, which are detailed in the following sub-sections. 

Relative Risk Reduction 

The BBN model is a probabilistic model. The probability of the occurrence of any causal variable (represented by a 
node) can be computed individually. This computation is typically performed for the end node (output node) in the 
network. For the AvSP models, the relative risk reduction of a given causal variable is a measure calculated by 
dividing the absolute risk reduction by the probability value from the baseline model. The absolute risk reduction is 
defined as the difference between the probability value from the baseline model and that from the model with 
inserted products. The relative risk reduction is a quantitative measure and very beneficial because it indicates the 
mitigation impact of the safety products. The relative risk reduction can be calculated to examine the individual 
contribution of a safety product or the combined contribution of a group of products (such as products of each TC or 
of each project) in reducing the target risk. For the FLAP model, both Automation Related Accident Rate node and 
Automation Related Incident Rate node are the outputs of interest. Therefore, the relative risk reductions on 
automation-related accident and incident of each notional product were computed and are shown in Exhibit 3. 
Similarly, the relative risk reductions on automation-related accident and incident per each project and all projects 
are presented in Exhibit 4. 


Exhibit 3. Notional Relative Risk Reduction per Product 


Relative Risk Reduction per Product 
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The exhibits suggest that, based on their placements, the products may have different accident and incident 
reduction impacts (e.g. VS ST-2 has the greatest impact on accident risk reduction among all products, whereas 
AEST-2 has equal impact on both accident and incident risk reduction). Note that the values below are generated by 
randomly assigned percentage impacts by product for illustration purposes. However, the actual product impact 
assessment is developed from the SME inputs by considering the status of the parent nodes as well as the presence 
of multiple products on the same node. It should also be noted that the risk reduction impact in the BBN propagates 
from the node at which the product is inserted to all downstream causal factor nodes. Because of the use of 
Bayesian calculus, risk reduction impacts computed from different groups of multiple products are not additive. For 
instance, the sum of individually-calculated products impact from three projects (i.e., 122.65%) is not equal to the 
integral products impact of all projects (i.e., 80.58%) as shown in Exhibit 4. 
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Exhibit 4. Notional Relative Risk Reduction per Project 


Relative Risk Reduction per Project 



Sensitivity Analysis 

The sensitivity analysis capability of the Hugin Software helps modelers and decision makers rank the causal factors 
for a highly concerned risk variable in the model. The software generates sensitivity values of all the network nodes 
with respect to an identified hypothesis variable in the baseline model. From a PA perspective, the amelioration of a 
sensitive causal factor (node) will have more impact in reducing the probability of the hypothesis variable when 
compared to a less sensitive node. In the FLAP model, the Flight Anomaly node is designated as the hypothesis 
variable since reduction of such anomalies inherently limits automation related accident and incidents. Note that the 
sensitivity values are not synonymous to probability values presented in the earlier sections. 

Exhibit 5. Normalized Causal Factor Sensitivity with respect to Flight Anomaly node 
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Exhibit 5 shows the top ten causal factor sensitivity values normalized against that of the Flight Anomaly 
node. The model suggests that the Regulatory Body node is the most sensitive node and can greatly impact the 
probability of a flight anomaly, and therefore, automation related mishaps. According to the preliminary results, the 
Regulatory Body node is a factor in only 23% of all accidents and incidents, however, due to its high influence on all 
downstream nodes (e.g. Manufacturer, Operator /Air line. Automation Design, etc.), it was determined to be the most 
sensitive node in affecting the outcome of the model. Also, the Automation Surprise node (with 65% probability) 
was identified as the second most sensitive node. Given its high sensitivity and probability, products that can reduce 
the probability of Automation Issues or its parent nodes inherently have a significant impact in lowering automation 
related accidents and incidents. The sensitivity ranking and individual causal factor occurrence probabilities present 
valuable information to decision makers from a program management point of view. Consequently, the information 
extracted from the model alone can be used to develop a balanced AvSP technology portfolio. 

Portfolio Gap Analysis 

Another outcome of accident models for technology assessment process is the ability to perform portfolio gap 
analysis. Following the insertion of AvSP products, the model nodes and notional products shown in Exhibit 2 are 
placed on a gap analysis matrix (Exhibit 6). 

Exhibit 6. Notional Gap Analysis 
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The matrix highlights the causal factors that are not addressed by the current AvSP portfolio (e.g. Flight 
Skills Degradation node), or portfolio elements that don’t have an impact on the modeled incident/accident (e.g. 
AEST-1 product). It also can help identify cases where too many portfolio elements are applied to a causal factor 
area that may not be the high sensitive or high probability accident and incident contributor. This information, along 
with the node sensitivity data given above provide decision makers with valuable knowledge in identifying novel 
technologies or areas in need of work and balancing the portfolio elements. 

Conclusions 

A well-balanced portfolio is crucial to program managers in order to achieve the project’s envisioned goals. NASA’s 
AvSP technology portfolio is assessed by SAIT using several qualitative and quantitative metrics. This paper 
illustrates one of the metrics that uses aviation safety models in BBN to gauge and rank the products’ impacts on 
reducing a predetermined safety risk. Besides the product impacts, the modeling effort alone also allows decision 
makers to identify the critical and high probability causal factors that affect the occurrence of an incident/accident. 
Coupled with the other metrics like cost, implementation risk, readiness level considered in the PA process, the 
modeling effort provides valuable information to program managers not only for AvSP but also potentially for other 
projects/programs within NASA as well as other organizations. 
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